This notebook contains a comparative analysis on the task of recognizing textual entailment between three approaches:
[1] Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy (2013): Learning Whom to Trust with MACE. In: Proceedings of NAACL-HLT 2013.
First we describe the task. Then, we apply the CrowdTruth metrics and identify clear and unclear example sentence. We then apply MACE. In the final part we perform two comparisons:
Data This noteboook uses the data gathered in the "Recognizing Textual Entailment" crowdsourcing experiment published in Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y. Ng: Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. EMNLP 2008, pages 254–263.
Task Description: Given two sentences, the crowd has to choose whether the second hypothesis sentence can be inferred from the first sentence (binary choice, true/false). Following, we provide an example from the aforementioned publication:
Text: “Crude Oil Prices Slump”
Hypothesis: “Oil prices drop”
A screenshot of the task as it appeared to workers can be seen at the following repository.
The dataset for this task was downloaded from the following repository, which contains the raw output from the crowd on AMT. Currently, you can find the processed input file in the folder named data. Besides the raw crowd annotations, the processed file also contains the text and the hypothesis that needs to be tested with the given text, which were given as input to the crowd.
In [1]:
#Read the input file into a pandas DataFrame
import pandas as pd
test_data = pd.read_csv("../data/rte.standardized.csv")
test_data.head()
Out[1]:
In [2]:
import crowdtruth
from crowdtruth.configuration import DefaultConfig
Our test class inherits the default configuration DefaultConfig, while also declaring some additional attributes that are specific to the Recognizing Textual Entailment task:
inputColumns: list of input columns from the .csv file with the input dataoutputColumns: list of output columns from the .csv file with the answers from the workerscustomPlatformColumns: a list of columns from the .csv file that defines a standard annotation tasks, in the following order - judgment id, unit id, worker id, started time, submitted time. This variable is used for input files that do not come from AMT or FigureEight (formarly known as CrowdFlower).annotation_separator: string that separates between the crowd annotations in outputColumnsopen_ended_task: boolean variable defining whether the task is open-ended (i.e. the possible crowd annotations are not known beforehand, like in the case of free text input); in the task that we are processing, workers pick the answers from a pre-defined list, therefore the task is not open ended, and this variable is set to Falseannotation_vector: list of possible crowd answers, mandatory to declare when open_ended_task is False; for our task, this is the list of relationsprocessJudgments: method that defines processing of the raw crowd data; for this task, we process the crowd answers to correspond to the values in annotation_vectorThe complete configuration class is declared below:
In [3]:
class TestConfig(DefaultConfig):
inputColumns = ["gold", "task", "text", "hypothesis"]
outputColumns = ["response"]
customPlatformColumns = ["!amt_annotation_ids", "orig_id", "!amt_worker_ids", "start", "end"]
# processing of a closed task
open_ended_task = False
annotation_vector = ["0", "1"]
def processJudgments(self, judgments):
# pre-process output to match the values in annotation_vector
for col in self.outputColumns:
# transform to lowercase
judgments[col] = judgments[col].apply(lambda x: str(x).lower())
return judgments
In [4]:
data, config = crowdtruth.load(
file = "../data/rte.standardized.csv",
config = TestConfig()
)
data['judgments'].head()
Out[4]:
In [5]:
results = crowdtruth.run(data, config)
The sentence metrics are stored in results["units"]. The uqs column in results["units"] contains the sentence quality scores, capturing the overall workers agreement over each sentences. The uqs_initial column in results["units"] contains the initial sentence quality scores, before appling the CrowdTruth metrics.
In [6]:
results["units"].head()
Out[6]:
In [7]:
# Distribution of the sentence quality scores and the initial sentence quality scores
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 15, 5
plt.subplot(1, 2, 1)
plt.hist(results["units"]["uqs"])
plt.ylim(0,270)
plt.xlabel("Sentence Quality Score")
plt.ylabel("#Sentences")
plt.subplot(1, 2, 2)
plt.hist(results["units"]["uqs_initial"])
plt.ylim(0,270)
plt.xlabel("Initial Sentence Quality Score")
plt.ylabel("# Units")
Out[7]:
The histograms above show that the final sentence quality scores are nicely distributed, with both lower and high quality sentences. We also observe that, overall, the sentence quality score increased after applying the CrowdTruth metrics, compared to the initial sentence quality scores.
The sentence quality score is a powerful measure to understand how clear the sentence is and the suitability of the sentence to be used as training data for various machine learning models.
The unit_annotation_score column in results["units"] contains the sentence-annotation scores, capturing the likelihood that an annotation is expressed in a sentence. For each sentence, we store a dictionary mapping each annotation to its sentence-annotation score.
In [8]:
results["units"]["unit_annotation_score"].head()
Out[8]:
First, we sort the sentence metrics stored in results["units"] based on the sentence quality score (uqs), in ascending order. Thus, the most clear sentences are found at the tail of the new structure:
In [9]:
sortedUQS = results["units"].sort_values(["uqs"])
sortedUQS = sortedUQS.reset_index()
We print the most clear unit, which is the last unit in sortedUQS:
In [10]:
sortedUQS.tail(1)
Out[10]:
The unit below is very clear because the text contains high overlap with the hypothesis. The relevance can be observed in the following parts of the hypothesis and of the text: "Pamplona fiesta has been celebrated for centuries" and "The centuries-old Pamplona fiesta".
In [11]:
print("Hypothesis: %s" % sortedUQS["input.hypothesis"].iloc[len(sortedUQS.index)-1])
print("Text: %s" % sortedUQS["input.text"].iloc[len(sortedUQS.index)-1])
print("Expert Answer: %s" % sortedUQS["input.gold"].iloc[len(sortedUQS.index)-1])
print("Crowd Answer with CrowdTruth: %s" % sortedUQS["unit_annotation_score"].iloc[len(sortedUQS.index)-1])
print("Crowd Answer without CrowdTruth: %s" % sortedUQS["unit_annotation_score_initial"].iloc[len(sortedUQS.index)-1])
We use the same structure as above and we print the most unclear unit, which is the first unit in sortedUQS:
In [12]:
sortedUQS.head(1)
Out[12]:
The unit below is very unclear because the text and the hypothesis contain overlapping words such as "1990" and "apartheid" and phrases that could be related, such as "South Africa" - "ANC" or "abolished" - "were to be lifted", but a clear relevance between the two can not be shown.
In [13]:
print("Hypothesis: %s" % sortedUQS["input.hypothesis"].iloc[0])
print("Text: %s" % sortedUQS["input.text"].iloc[0])
print("Expert Answer: %s" % sortedUQS["input.gold"].iloc[0])
print("Crowd Answer with CrowdTruth: %s" % sortedUQS["unit_annotation_score"].iloc[0])
print("Crowd Answer without CrowdTruth: %s" % sortedUQS["unit_annotation_score_initial"].iloc[0])
The worker metrics are stored in results["workers"]. The wqs columns in results["workers"] contains the worker quality scores, capturing the overall agreement between one worker and all the other workers. The wqs_initial column in results["workers"] contains the initial worker quality scores, before appling the CrowdTruth metrics.
In [14]:
results["workers"].head()
Out[14]:
In [15]:
# Distribution of the worker quality scores and the initial worker quality scores
plt.rcParams['figure.figsize'] = 15, 5
plt.subplot(1, 2, 1)
plt.hist(results["workers"]["wqs"])
plt.ylim(0,50)
plt.xlabel("Worker Quality Score")
plt.ylabel("#Workers")
plt.subplot(1, 2, 2)
plt.hist(results["workers"]["wqs_initial"])
plt.ylim(0,50)
plt.xlabel("Initial Worker Quality Score")
plt.ylabel("#Workers")
Out[15]:
The histograms above shows the worker quality scores and the initial worker quality scores. We observe that the worker quality scores are distributed across a wide spectrum, from low to high quality workers. Furthermore, the worker quality scores seem to have a more normal distribution after computing the CrowdTruth iterations, compared to the initial worker quality scores.
Low worker quality scores can be used to identify spam workers, or workers that have misunderstood the annotation task. Similarly, high worker quality scores can be used to identify well performing workers.
The annotation metrics are stored in results["annotations"]. The aqs column contains the annotation quality scores, capturing the overall worker agreement over one annotation.
In [16]:
results["annotations"]
Out[16]:
In the dataframe above we observe that after iteratively computing the sentence quality scores and the worker quality scores the overall agreement on the annotations increased. This can be seen when comparing the annotation quality scores with the initial annotation quality scores.
We first pre-processed the crowd results to create compatible files for running the MACE tool. Each row in a csv file should point to a unit in the dataset and each column in the csv file should point to a worker. The content of the csv file captures the worker answer for that particular unit (or remains empty if the worker did not annotate that unit).
The following implementation of MACE has been used in these experiments: https://github.com/dirkhovy/MACE.
In [17]:
# MACE input file sample
import numpy as np
mace_test_data = pd.read_csv("../data/mace_rte.standardized.csv", header=None)
mace_test_data = test_data.replace(np.nan, '', regex=True)
mace_test_data.head()
Out[17]:
For each sentence and each annotation, MACE computes the sentence annotation probability score, which shows the probability of each annotation to be expressed in the sentence. MACE sentence annotation probability score is similar to the CrowdTruth sentence-annotation score.
In [18]:
# MACE sentence annotation probability scores:
import pandas as pd
mace_data = pd.read_csv("../data/results/mace_units_rte.csv")
mace_data.head()
Out[18]:
For each worker in the annotators set we have MACE worker competence score, which is similar to the CrowdTruth worker quality score.
In [19]:
# MACE worker competence scores
mace_workers = pd.read_csv("../data/results/mace_workers_rte.csv")
mace_workers.head()
Out[19]:
We read the worker quality scores as returned by CrowdTruth and MACE and merge the two dataframes:
In [20]:
mace_workers = pd.read_csv("../data/results/mace_workers_rte.csv")
crowdtruth_workers = pd.read_csv("../data/results/crowdtruth_workers_rte.csv")
workers_scores = pd.merge(mace_workers, crowdtruth_workers, on='worker')
workers_scores = workers_scores.sort_values(["wqs"])
workers_scores.head()
Out[20]:
Plot the quality scores of the workers as computed by both CrowdTruth and MACE:
In [21]:
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
plt.scatter(
workers_scores["competence"],
workers_scores["wqs"],
)
plt.plot([0, 1], [0, 1], 'red', linewidth=1)
plt.title("Worker Quality Score")
plt.xlabel("MACE")
plt.ylabel("CrowdTruth")
Out[21]:
In the plot above we observe that MACE favours extreme values, which means that the identified low quality workers will have very low scores, e.g., below 0.2 and the best workers will have quality scores of 1.0, or very close to 1.0. On the other side, CrowdTruth has a smaller interval of values, starting from around 0.25 to 0.9.
Following, we compute the correlation between the two values using Spearman correlation and Kendall's tau correlation, to see whether the two values are correlated. More exactly, we want to see whether, overall, both metrics identify as low quality or high quality similar workers, or they are really divergent in their outcome.
In [22]:
from scipy.stats import spearmanr
x = workers_scores["wqs"]
x_corr = workers_scores["competence"]
corr, p_value = spearmanr(x, x_corr)
print ("correlation: ", corr)
print ("p-value: ", p_value)
Spearman correlation shows shows a strong to very strong correlation between the two computed values, and the correlation is significant. This means that overall, even if the two metrics provide different values, they are indeed correlated and low quality workers receive low scores and high quality workers receive higher scores from both aggregation methods.
In [23]:
from scipy.stats import kendalltau
x1 = workers_scores["wqs"]
x2 = workers_scores["competence"]
tau, p_value = kendalltau(x1, x2)
print ("correlation: ", tau)
print ("p-value: ", p_value)
Even with Kendall's tau rank correlation, we observe a moderate to strong correlation between the two computed values, where the correlation is significant.
Further, we compute the difference of the two quality scores and we check one worker for which the difference is very high.
In [24]:
workers_scores["diff"] = workers_scores["wqs"] - workers_scores["competence"]
workers_scores = workers_scores.sort_values(["diff"])
workers_scores.tail(5)
Out[24]:
We take for example the worker with the id "A2QPX2MS844TYJ" and check the overall disagreement among the workers on the units annotated by them. MACE rated the worker with a quality score of 0.01 while CrowdTruth rated the worker with a quality score of 0.43
What we observe in the dataframe below, where we show the units annotated by the worker "A2QPX2MS844TYJ", is that the worker "A2QPX2MS844TYJ" annotated, in general, units with high disagreement, i.e., which are not very clear. While MACE marked the worker as low quality because it seems that they always picked the same answer, CrowdTruth also considered the difficulty of the units, and thus, giving it a higher weight.
In [25]:
# Sample of sentences annotated by worker A2QPX2MS844TYJ
units = list(test_data[test_data["!amt_worker_ids"] == "A2QPX2MS844TYJ"]["orig_id"])
units_df = sortedUQS[sortedUQS["unit"].isin(units)]
units_df = units_df.sort_values(["uqs_initial"])
units_df
Out[25]:
Next, we look into the crowd performance in terms of F1-score compared to expert annotations. We compare the crowd performance given the three aggregation methods: CrowdTruth, MACE and Majority Vote. We read the result files as given by MACE and CrowdTruth.
In [26]:
import pandas as pd
import numpy as np
mace = pd.read_csv("../data/results/mace_units_rte.csv")
crowdtruth = pd.read_csv("../data/results/crowdtruth_units_rte.csv")
The following two functions compute the F1-score of the crowd compared to the expert annotations. The first function computes the F1-score at every sentence-annotation score threshold. The second function computes the F1-score for the majority vote approach, i.e., when at least half of the workers picked the answer.
In [27]:
def compute_F1_score(dataset, label, gold_column, gold_value):
nyt_f1 = np.zeros(shape=(100, 2))
for idx in xrange(0, 100):
thresh = (idx + 1) / 100.0
tp = 0
fp = 0
tn = 0
fn = 0
for gt_idx in range(0, len(dataset.index)):
if dataset[label].iloc[gt_idx] >= thresh:
if dataset[gold_column].iloc[gt_idx] == gold_value:
tp = tp + 1.0
else:
fp = fp + 1.0
else:
if dataset[gold_column].iloc[gt_idx] == gold_value:
fn = fn + 1.0
else:
tn = tn + 1.0
nyt_f1[idx, 0] = thresh
if tp != 0:
nyt_f1[idx, 1] = 2.0 * tp / (2.0 * tp + fp + fn)
else:
nyt_f1[idx, 1] = 0
return nyt_f1
def compute_majority_vote(dataset, label, gold_column, gold_value):
tp = 0
fp = 0
tn = 0
fn = 0
for j in range(len(dataset.index)):
if dataset[label].iloc[j] >= 0.5:
if dataset[gold_column].iloc[j] == gold_value:
tp = tp + 1.0
else:
fp = fp + 1.0
else:
if dataset[gold_column].iloc[j] == gold_value:
fn = fn + 1.0
else:
tn = tn + 1.0
return 2.0 * tp / (2.0 * tp + fp + fn)
F1-score for the annotation "1" or "true":
In [28]:
F1_crowdtruth = compute_F1_score(crowdtruth, "true", "gold", 1)
print("Best CrowdTruth F1 score for annotation 'true': ", F1_crowdtruth[F1_crowdtruth[:,1].argsort()][-1:])
F1_mace = compute_F1_score(mace, "true", "gold", 1)
print("Best MACE F1 score for annotation 'true': ", F1_mace[F1_mace[:,1].argsort()][-1:])
F1_majority_vote = compute_majority_vote(crowdtruth, 'true_initial', "gold", 1)
print("Majority Vote F1 score for annotation 'true': ", F1_majority_vote)
From the results above we observe that MACE and CrowdTruth perform very close to each other, and they both perform much better than Majority Vote. As we can observe in the overall initial sentence quality score, there are many unclear sentences in the dataset where half of the workers picked "true" as an answer and half as "false" (more than 220 examples out of 800). In this case, the Majority Vote approach is not desirable since picking the right answer might not be better than throwing a coin. However, approaches that account for unit difficulty and workers quality (such as CrowdTruth and MACE) are more suitable.
To further explore the CrowdTruth and MACE quality metrics, download the aggregation results in .csv format for: